Information processing system, information processing apparatus, output apparatus, program, and recording medium

ABSTRACT

An information processing apparatus provided with a display portion including a plurality of sub-pixels every pixel, connected to an output apparatus for presenting an image displayed on the display portion to a user through a lens, and producing the image to be displayed on the display portion receives source image data becoming an object of display, obtains a corresponding pixel on the source image data by information on a position of the sub-pixel and a predetermined conversion expression every sub-pixel included in a pixel with respect to the respective pixels constituting corrected image data, and determines luminances of the respective sub-pixels based on a pixel value of the pixel on the source image obtained every sub-pixel, thereby producing the corrected image data.

TECHNICAL FIELD

The present invention relates to an information processing system, aninformation processing apparatus, an output apparatus, a program, and arecording medium.

BACKGROUND ART

In recent years, for example, like a head-mounted display or a smartglass, there has been an output apparatus which a user mounts on his/herhead to use. Such an output apparatus is provided with a pair ofdisplays which are individually disposed in front of the left and righteyes of the user, and presents images displayed on the displays to theuser. Since such an output apparatus enables the user to visuallyrecognize a picture having realistic sensation, the output apparatus isutilized in a virtual reality technique and the like.

SUMMARY Technical Problems

With such an output apparatus, in order to widen a viewing anglethereof, images displayed on the displays can be dared to be formed infront of the eyes of the user through lenses each having a strongdistortion. When such lenses are used, the images displayed on thedisplays are not the images themselves presented to the user. That is tosay, corrected images for which a distortion reverse to a lensdistortion is previously given to the original images (source images)are produced, and the corrected images are displayed. As a result, theimages which are visually recognized by the user through the lensesbecome ones corresponding to the source images.

Here, the display is generally constituted by a plurality of pixels. Inthis case, each of the pixels is further constituted by three sub-pixelswhich emit light of three primary colors: red (R); green (G); and blue(B). However, heretofore, although the positions of the respectivesub-pixels have been dispersed within a surface which one pixeloccupies, the correction has been carried out on the assumption thateach of the colors of R, G, and B is emitted at one predetermined pointwithin the pixel (at the center of the surface which the pixel normallyoccupies).

On the other hand, as the viewing angle is further increased, the numberof pixels per unit viewing angle is reduced. Therefore, even when thereis a slight error of the correction, this affects the image ofappearance. For example, in the pixel closer to the peripheral portionof the display, a slight difference in position on the display exerts alarge influence on a difference in corresponding position on the sourceimage. Such an error in position on the display (the difference in pixelvalue due to reference to a false pixel on the source image) becomes aproblem in terms of a color aberration.

The present invention has been made in the light of the actual situationdescribed above, and one of objects thereof is to provide an informationprocessing system, an information processing apparatus, an outputapparatus, a program, and a recording medium each of which is capable ofcarrying out correction of an image in which a sub-pixel is taken intoaccount.

Solution to Problems

The present invention for solving the problems of the past exampledescribed above is an information processing system. The informationprocessing system includes an output apparatus and an informationprocessing apparatus. The output apparatus is provided with a displayportion including a plurality of sub-pixels every pixel, and presents animage displayed on the display portion to a user through a lens. Theinformation processing apparatus is connected to the output apparatusand produces an image to be displayed on the display portion. Theinformation processing apparatus includes means for receiving sourceimage data becoming an object of display, correction means, and outputmeans. The correction means produces corrected image data for which adistortion of a lens provided in the output apparatus is correctedthrough a predetermined conversion expression based on the source imagedata received. The correction means obtains corresponding pixels on thesource image data through an arithmetic operation by information on aposition of a sub-pixel every sub-pixel and the conversion expressionincluded in the pixel with respect to the respective pixels constitutingcorrected image data, and produces the corrected image data bydetermining luminances of the respective sub-pixels based on a pixelvalue of the pixel on the source image obtained every sub-pixel. Theoutput means outputs the corrected image data produced to the outputapparatus.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a configuration block diagram representing an example of aninformation processing system according to an embodiment of the presentinvention.

FIG. 2 is a configuration block diagram representing an example of anoutput apparatus according to the embodiment of the present invention.

FIG. 3 depicts explanatory diagrams representing an example ofarrangement positions of sub-pixels of a display portion of the outputapparatus according to the embodiment of the present invention.

FIG. 4 is a functional block diagram representing an example of aninformation processing apparatus according to the embodiment of thepresent invention.

FIG. 5 is a flow chart representing a processing example of theinformation processing apparatus according to the embodiment of thepresent invention.

DESCRIPTION OF EMBODIMENT

An embodiment of the present invention will be described with referenceto the drawings. An information processing system 10 according to theembodiment of the present invention, as exemplified in FIG. 1, includesan information processing apparatus 11 and an output apparatus 12.

In an example of the present embodiment, the information processingapparatus 11, for example, is computer-based equipment such as ahome-use video game system. Specifically, as depicted in FIG. 1, theinformation processing apparatus 11 includes a control portion 21, amemory portion 22, and an interface portion 23. In addition, the outputapparatus 12, for example, is a head-mounted display. As schematicallyexemplified in FIG. 2, the output apparatus 12 includes a controlportion 31, a memory portion 32, an input/output (I/O) interface portion33, a pair of display portions 34L and 34R, and a pair of lenses 35L and35R.

Here, the control portion 21 of the information processing apparatus 11is a program-controlled device such as a central processing unit (CPU)which operates in accordance with a program stored in the memory portion22, or a graphics processing unit (GPU) as an image processing module.The control portion 21 of the present embodiment receives source imagedata becoming an object of display in the output apparatus 12, andproduces corrected image data which is obtained by correcting adistortion of the lens 35 included in the output apparatus 12 through apredetermined conversion expression based on the source image data thusreceived. In addition, the control portion 21 outputs the resultingcorrected image data to the output apparatus 12. An operation of thecontrol portion 21 will be described later in detail.

The memory portion 22 is a memory device, a disc device or the like. Aprogram which is to be executed by the control portion 21 is stored inthe memory portion 22. This program may be one which is stored in acomputer readable and non-transitory recording medium such as a digitalversatile disc read-only memory (DVD-ROM) in order to be presented, andmay be stored in the memory portion 22. In addition, the memory portion22 may operate as a work memory as well of the control portion 21, and aframe buffer area in which image data becoming an object of processingis stored, for example, may be set in the memory portion 22.

The interface portion 23, for example, is a high-definition multimediainterface (HDMI) (registered trademark) port, a universal serial bus(USB) port or the like, and is connected to the output apparatus 12. Inaddition, the interface portion 23 outputs data (such as image data) tothe output apparatus 12 in accordance with an instruction inputtedthereto from the control portion 21. In addition, the interface portion23 outputs information inputted thereto from the output apparatus 12 tothe control portion 21.

The control portion 31 of the output apparatus 12 is aprogram-controlled device, such as a CPU, which operates in accordancewith the program stored in the memory portion 32. In the presentembodiment, the control portion 31 executes processing for outputtingcorrected image data which is inputted from the information processingapparatus 11 through the I/O interface portion 33 to the display portion34. In addition, the control portion 31 may execute processing forreceiving a request for the data stored in the memory portion 32 fromthe outside, and reading out the requested data from the memory portion32 to send the requested data thus read out to a request source. Anoperation of the control portion 31 will be described later.

The memory portion 32 is a memory device or the like, and a programwhich is to be executed by the control portion 31 is stored in thememory portion 32. This program may be one which is stored in a computerreadable and non-transitory recording medium such as a DVD-ROM in orderto be presented, and may be stored in the memory portion 32. Inaddition, the memory portion 32 may operate as a work memory as well ofthe control portion 31, and, for example, an area in which the correctedimage data for display is to be stored may be set in the memory portion32.

The I/O interface portion 33, for example, is an HDMI (registeredtrademark) port, a USB port or the like, and is connected to theinformation processing apparatus 11 and the like.

A pair of display portions 34L and 34R, for example, is a liquid crystaldisplay or the like, and, as exemplified in FIG. 2, is provided so as tocorrespond to each of the left and right eyes of the user (hereinafter,collectively written as the display portion 34 when it is unnecessary tomake a distinction). Each of the display portions 34 includes aplurality of pixels 341. Each of the pixels 341 is further provided withsub-pixels 342 which emit light of R, light of G, and light of B. Thesub-pixels 342 are arranged in positions as exemplified in FIG. 3(a) orFIG. 3(b) every pixel 341. In this example, in this case, a center of asub-pixel 342 r of R is located in a position (Cx+Rx, Cy+Ry) which is(Rx, Ry) apart from a central coordinate (Cx, Cy) of the pixel 342 inboth FIG. 3(a) and FIG. 3(b). In addition, a center of a sub-pixel 342 gof G is also located in a position (Cx+Gx, Cy+Gy) which is (Gx, Gy)apart from the central coordinate (Cx, Cy) of the pixel 342 in both FIG.3(a) and FIG. 3(b). On the other hand, a center of a sub-pixel 342 b ofB is located in a position (Cx+Bx1, Cy+By1) which is (Bx1, By1) apartfrom the central coordinate (Cx, Cy) of the pixel 342 in FIG. 3(a). InFIG. 3(b), the center of the sub-pixel 342 b of B is located in aposition (Cx+Bx2, Cy+By2) which is (Bx2, By2) (Bx1=Bx2, By1 By2 in theexamples of FIGS. 3(a) and 3(b)) apart from the central coordinate (Cx,Cy) of the pixel 342. It should be noted that here, the centralcoordinate is determined as a point within either the pixel 341 or thesub-pixel 342 in advance like a point at which diagonals of a boundingrectangle of either the pixel 341 or the sub-pixel 342 cross each other.

A pair of lenses 35L and 35R is arranged between the correspondingdisplay portions 34L and 34R, and the eyes of the user who sees theimages displayed on the display portions 34L and 34R concerned(hereinafter, collectively written as the lens 35 when it is unnecessaryto make a distinction). The lens 35 causes a predetermined distortion(for example, a pincushion distortion which contracts in a pincushionshape toward a central portion when a grid surface is displayed) to begenerated in an image (an image corresponding to the corrected imagedata) displayed on the corresponding display portion 34, and causes theeye of the user to recognize the resulting image.

Here, a description will be given with respect to an operation of thecontrol portion 21 of the information processing apparatus 11. Thecontrol portion 21, as exemplified functionally in FIG. 4, includes asource image receiving portion 41, a corrected image producing portion42, and an output portion 42. Here, the source image receiving portion41 receives inputs of a pair of source image data for the left eye andthe right eye. In an example of the present embodiment, the controlportion 21 executes processing as well (other pieces of processing),such as processing for a game, different from the correction of theimage in parallel (in a time division manner). Thus, the source imagereceiving portion 41 may also receive the source image data from theother pieces of processing concerned.

The corrected image producing portion 42 subjects each of a pair ofsource image data to conversion determined by a predetermined conversionexpression (for example, an arithmetic expression of conversion forproviding a barrel type distortion) for cancelling the distortion of thelens 35, and produces the corrected image data. Specifically, withrespect to each of a pair of the corrected image data for the left eyeand the right eye, the corrected image producing portion 42 obtains thepixels on the source image data corresponding to the sub-pixels throughan arithmetic operation by the information on the positions of thesub-pixels and the conversion expression described above every sub-pixelincluded in the respective pixels constituting the corrected image data.Then, the corrected image producing portion 42 determines luminances ofthe respective sub-pixels based on the pixel values of the pixels on thesource image concerned obtained every sub-pixel, thereby producing thecorrected image data. The output portion 43 outputs the corrected imagedata produced in the corrected image producing portion 42.

Next, a description will be given with respect to an operation of thecorrected image producing portion 42. The corrected image producingportion 42 executes next processing for producing the corrected imagedata with respect to each of a pair of source image data. When thecorrected image producing portion 42 starts the processing for producingthe corrected image data with respect to any of the source image data,as exemplified in FIG. 5, the corrected image producing portion 42secures an area in which the corrected image data is to be stored in thememory portion 22 (S1). This area is secured in correspondence to thenumber of pixels in the display portion 34 of the output apparatus 12.

The corrected image producing portion 42 selects one of the pixels 341of the corrected image data (S2). It is only necessary that thisselection, for example, is carried out in the order of scanning linefrom the pixel in a top-left corner. In other words, it is onlynecessary that the pixels are successively selected from the left to theright in a row of the pixels located in the uppermost side, and next,the selection is shifted for next row therebelow (one row therebelow),and the pixels are successively selected from the left to the rightagain, and so forth. Of course, the selection of the pixels is by nomeans limited to this order, and the pixels may be selected in thearbitrary order. In addition, by utilizing a GPU, a plurality of pixelsis selected in parallel to execute next processing.

When the corrected image producing portion 42 selects one of the pixels341 corresponding to the corrected image data, the corrected imageproducing portion 42 successively selects the sub-pixels 342 included inthe pixel thus selected (S3). Specifically, it is only necessary toselect the sub-pixels in a predetermined order such as an order of R, G,and B. The corrected image producing portion 42 calculates the positionof the central coordinate of the sub-pixel 342 selected in theprocessing S3 by using the central coordinate (Cx, Cy) of the pixel 341selected in the processing S2 (S4).

If the arrangements of the sub-pixels 342 are identical to one anotherin all the pixels 341, the central coordinate of the sub-pixel 342 canbe calculated by referring to information which represents how much farthe central coordinate of each of the sub-pixels 342 is located apartfrom the central coordinate of the pixel 341 and is stored in the memoryportion 22 in advance.

In addition, when the central coordinate of at least one sub-pixel 342is different depending on the pixels 341 (for example, as exemplified inFIGS. 3(a) and 3(b), when there are two kinds of arrangements, and soforth), the information representing how much far the central coordinateof each of the sub-pixels 342 is located apart from the centralcoordinate of the pixel 341 is stored in the memory portion 22. Inaddition, information specifying the pixel 341 and informationrepresenting which kind of arrangement the sub-pixel 342 takes in thepixel 341 specified by that information are stored in the memory portion22 with being assembled with each other every pixel 341. The correctedimage producing portion 42 specifies the kind of arrangement in thepixel 341 selected in the processing S2 based on the informationconcerned. In addition, the corrected image producing portion 42calculates the positions of the central coordinates of the respectivesub-pixels 342 by referring to the information (the information on theposition of the sub-pixel 342 representing how much far the centralcoordinate of the sub-pixel 342 is located apart from the centralcoordinate of the pixel 341) on the central coordinates of therespective sub-pixels 342 in the arrangement having the kind thusspecified.

Moreover, when the central coordinate of at least one sub-pixel 342 isdifferent depending on the pixel 341 (for example, as exemplified inFIGS. 3(a) and 3(b), when there are the two kinds of arrangements, andso forth), the information representing how much far the centralcoordinate of each of the sub-pixels 342 is located apart from thecentral coordinate of the pixel 341 with respect to each of the pixels341 may be averaged to obtain the average central coordinate of therespective sub-pixels 342. The resulting average central coordinate ofthe sub-pixel 342 may be stored in the memory portion 22 in advance. Inthis case, the corrected image producing portion 42 reads out theinformation (average information) representing how much far it islocated apart from the central coordinate of the pixel 341 selected inthe processing S2 from the memory portion 22 and refers to thatinformation, thereby calculating the positions of the centralcoordinates of the respective sub-pixels 342.

The corrected image producing portion 42 obtains the correspondingcoordinate on the source image data, corresponding to the centralcoordinate of the sub-pixel 342 through the arithmetic operation,calculated in the processing S4 (S5). With that arithmetic operation,the coordinate on the source image data which is converted into thecentral coordinate of the sub-pixel 342 selected by a predeterminedconversion expression is obtained as the corresponding coordinatethrough the arithmetic operation by using the predetermined conversionexpression, cancelling the distortion of lens 35, for obtaining thecorrected image data from the source image data, and the centralcoordinate of the sub-pixel 342 calculated in the processing S4. Sincethe concrete calculating method is generally known, a detaileddescription thereof is omitted herein.

The corrected image producing portion 42 determines the luminance of thesub-pixel 342 selected in the processing S3 based on the pixel values(the component of the pixel value corresponding to the color of thesub-pixel 342 selected in the processing S3) in the pixel (there may bea plurality of pixels) falling within the predetermined range on thesource image data with the corresponding coordinate on the source imagedata obtained as the center (S6). In general, the value of thecorresponding coordinate obtained from the processing S5 is not locatedat the center of any of the pixels. Therefore, the determination of theluminance herein is obtained by interpolating the pixel values in thepixels falling within the predetermined range with the resultingcorresponding coordinate on the source image data as the center. Theinterpolations, which are generally known, such as nearest neighborinterpolation, bilinear interpolation, and bicubic interpolation can beused as this interpolation method. As an example, the arithmeticoperation relating to the processing S5 and S6 can be carried out byutilizing lens distortion correcting processing using software such asan Open CV.

The corrected image producing portion 42 repeatedly executes theprocessing from S3 to S6 with respect to the respective sub-pixels 342.As a result, the luminances of the sub-pixels 342 included in the pixel341 selected in the processing S2 are individually determined, and thepixel value of the pixel 341 concerned is determined. That is to say,one of the features in the present embodiment is such that the correctedimage data obtained by converting the source image data is obtainedbased on the position within the pixel not every pixel, but everysub-pixel.

The corrected image producing portion 42 repeatedly executes theprocessing from the processing S2 to the processing S6 with respect tothe respective pixels 341. As a result, the corrected image producingportion 42 successively obtains the pixel values, of the respectivepixels 341, contained in the corrected image data. In addition, thecorrected image producing portion 42 executes the processing describedabove with respect to the left eye and the right eye, thereby producinga pair of corrected image data for the left eye and the right eye.

Processing in which Assembly Error of Lenses is Taken into Consideration

In addition, in an example of the present embodiment, the correctedimage producing portion 42 may further obtain information on an assemblyerror of a pair of display portions 34L and 34R and a pair of lenses 35with which the output apparatus 12 is provided, and may further executeprocessing for moving the corrected image data in parallel based on thisinformation.

That is to say, for the output apparatus 12 of the present embodiment,the assembly error of the display portions 34L and 34R (hereinafter,collectively written as the display portion 34 when it is unnecessary totake distinction), and the lenses 35 (corresponding to each of the lens35L on the left eye side and the lens 35R on the right eye side) ismeasured at the time of factory shipment. The measurement can be carriedout by using the generally known method. Specifically, this measurementis carried out as follows. Firstly, after the display portions 34L and34R on which the image for the left eye and the image for the right eyeare to be individually displayed, and the lenses 35 are assembled,images (markers) of a pair of line segments (crosses) which intersecteach other in positions of the centers of the display portions 34L and34R are displayed at the centers of the display portions 34L and 34R.

Next, the markers, displayed on the display portions 34L and 34Rindividually, are photographed by using a camera for measurement set ina state in which central axes of the ideal left and right assemblypositions of the display portions 34L and 34R and the lenses 35L and35R, and optical axes thereof are aligned with each other.

Then, how much far the images of the markers thus photographed areshifted from the position corresponding to the optical axis of thecamera for measurement is measured.

As a result, the position shift (represented by the coordinate of thedisplay portion 34) between the images displayed on the correspondingdisplay portions 34L and 34R visually recognized through the lenses 35Land 35R, and a point of view is measured. Then, the measurement results(the measurement results about the sets of lens 35L and display portion34L on the left eye side, and lens 35R and display portion 34R on theright eye side) are stored as information on the assembly error in thememory portion 32.

In addition, the control portion 31 of the output apparatus 12 receivesa request for the information on the assembly error from the outside(such as the information processing apparatus 11). Then, in response tothe request, the control portion 31 reads out the information on theassembly error stored in the memory portion 32, and outputs theinformation on the assembly error thus read out to the informationprocessing apparatus 11 or the like as the request source.

In an example of the present embodiment, by configuring the outputapparatus 12 in such a manner, in the information processing apparatus11, the corrected image producing portion 42 sends the request for theinformation on the assembly error to the output apparatus 12. Then, thecorrected image producing portion 42 receives the information on theassembly error which the output apparatus 12 outputs in response to therequest concerned, and stores the information on the assembly error inthe memory portion 22.

Then, the corrected image producing portion 42 reads out the informationon the assembly error about the lens 35L and the display portion 34L onthe left eye side from the memory portion 22, and moves in parallel thecorrected image data on the left eye by an amount corresponding to theinformation on the error thus read out. That is to say, the correctedimage producing portion 42 moves in parallel the whole of the correctedimage data for the left eye in such a way that the pixel to be displayedat the center of the display portion 34L of the corrected image databecomes in position the pixel of the display portion 34L on the opticalaxis of the line of sight of the user visually recognized through thelens 35L. Thus, the corrected image producing portion 42 obtains secondcorrected image data for the left eye. In addition, the corrected imageproducing portion 42 reads out the information on the assembly errorabout the lens 35R and the display portion 34R on the right eye sidefrom the memory portion 22, and moves in parallel the corrected imagedata on the left eye by an amount corresponding to the information onthe error thus read out. In other words, the corrected image producingportion 42 moves in parallel the whole of the corrected image data forthe right eye in such a way that the pixel to be displayed at the centerof the display portion 34R of the corrected image data becomes inposition the pixel of the display portion 34R on the optical axis of theline of sight of the user visually recognized through the lens 35R.Thus, the corrected image producing portion 42 obtains second correctedimage data for the right eye.

Incidentally, the corrected image data which is obtained through theconversion from the source image data based on the position within thepixel for each sub-pixel is further moved in parallel herein. However,the corrected image data which is obtained through the conversion fromthe source image data based on the positions of the centers of therespective pixels may also be moved in parallel.

In addition, although the processing for the parallel movement isexecuted after the corrected image data is obtained herein, theprocessing for the parallel movement may also be executed together inthe process of the processing for producing the corrected image data.

Operation

The information processing system of the present embodiment includes theconfiguration described above, and operates as follows. The controlportion 21 of the information processing apparatus 11 secures the areain which the corrected image data individually corresponding to a pairof source image data for the left eye and the right eye which isproduced in the processing for a game is to be stored in the memoryportion 22.

The control portion 21 executes the following processing with respect toeach of a pair of corrected image data. That is to say, the controlportion 21 successively selects the sub-pixels within the pixelcontained in the corrected image data. The control portion 21 calculatesthe central coordinate of the sub-pixel thus selected. Incidentally, inthe case where the central coordinates of at least a part of sub-pixelsare different depending on the pixels (for example, in the case where asexemplified in FIGS. 3(a) and 3(b), there are the two kinds ofarrangements, and so forth), the information representing how much farthe central coordinates of the respective sub-pixels are located apartfrom the central coordinate of the pixel is averaged with respect to thepixels, and the average central coordinates of the respective sub-pixelsare obtained. Then, the resulting average central coordinates are used.

The control portion 21 obtains the coordinate on the source image datawhich is to be converted into the central coordinate of the sub-pixelconcerned by the conversion expression for cancelling the distortion oflens 35 as the corresponding coordinate through the arithmeticoperation. The luminance of the selected sub-pixel is determined basedon the pixel value (a component of the pixel value corresponding to thecolor of the selected pixel) in the pixel (there may be a plurality ofpixels) falling within the predetermined range with the correspondingcoordinate concerned as the center on the source image data. The controlportion 21 executes this processing either every sub-pixel or everypixel, thereby obtaining the luminance (pixel value) for each sub-pixelof the pixels contained in the corrected image data. In addition, thecontrol portion 21 executes these pieces of processing with respect tothe processing for the left eye and the processing for the right eye,thereby producing a pair of corrected image data for the left eye andthe right eye.

The control portion 21 outputs a pair of corrected image data thusproduced to the output apparatus 12 through the interface portion 23. Inthe output apparatus 12, the corrected image data for the left eye thusinputted is displayed on the display portion 34L for the left eye, andthe corrected image data for the right eye thus inputted is displayed onthe display portion 34R for the right eye.

These pieces of corrected image data thus displayed are obtained throughsuch correction as to cancel the distortion of the lens 35 as describedabove. These pieces of corrected image data thus displayed suffer thedistortion of the lens 35 through the corresponding lenses 35L and 35R,so that the deformation by the correction and the deformation by thedistortion cancel each other. Thus, the images corresponding to thesource image data enter the eyes of the user.

In the corrected image data produced by the information processingapparatus 11 of the present embodiment, when the arrangement of thesub-pixels, for example, are one depicted in FIG. 3(a) or FIG. 3(b),since the sub-pixels of R and G are arranged in the longitudinaldirection, the sub-pixels of R and G are asymmetric with respect to ahorizontal line. Therefore, when a pair of images which is symmetricwith respect to the horizontal line is set as the source image data, thecorrected image data obtained therefrom becomes the results differentfrom each other.

Moreover, the control portion 21 of the information processing apparatus11 may execute the processing in which the assembly error of the lensesis taken into consideration. In this case, the control portion 21 sendsthe request for the information on the assembly error to the outputapparatus 12 at the time of production of the corrected image data. Thecontrol portion 21 receives the information on the assembly error outputfrom the output apparatus 12 in response to the request concerned, andstores the information concerned in the memory portion 22.

Then, the control portion 21 reads out the information on the assemblyerror with respect to the lens 35L and the display portion 34L on theleft eye side from the memory portion 22, and moves in parallel thecorrected image data for the left eye by the amount corresponding to theinformation on the error thus read out to obtain the second correctedimage data for the left eye. In addition, the control portion 21 readsout the information on the assembly error with respect to the lens 35Rand the display portion 34R on the right eye side from the memoryportion 22, and moves in parallel the corrected image data for the righteye by the amount corresponding to the information on the error thusread out to obtain the second corrected image data for the right eye.

Then, the control portion 21 outputs a pair of second corrected imagedata thus produced to the output apparatus 12 through the interfaceportion 23. In the output apparatus 12, the second corrected image datafor the left eye thus inputted is displayed on the display portion 34Lfor the left eye, and the second corrected image data for the right eyethus inputted is displayed on the display portion 34R for the right eye.

These pieces of second corrected image data thus displayed are obtainedthrough such correction as to cancel the distortion of the lens 35 asdescribed above. These pieces of second corrected image data thusdisplayed suffer the distortion of the lens 35 through the correspondinglenses 35L and 35R, so that the deformation by the correction and thedeformation by the distortion cancel each other. Thus, the imagescorresponding to the source image data enter the eyes of the user. Inaddition, in this example, even if the display portion 34 and the lens35 are arranged off the desired positions at the time of the assembly,the image which is displayed on the display portion 34 is corrected insuch a way that the center of the original source image data is locatedon the optical axis of the line of sight of the user through the lens35. Therefore, it is possible to provide the intended sense ofperspective for the user.

Modified Changes

In the present embodiment, the information on the positions of thesub-pixels within the respective pixels in the display portion 34 of theoutput apparatus 12 is stored in the memory portion 22. However, theinformation on the positions of the sub-pixels, for example, may also bestored at the time of installation of the driver of the output apparatus12.

In another example of the present embodiment, when the output apparatus12 is connected, the information processing apparatus 11 may acquire theinformation on the positions of the sub-pixels from the output apparatus12, and may store the information on the positions of the sub-pixels inthe memory portion 22. In this example, the information (either theinformation representing how much far the central coordinates of therespective sub-pixels 342 are located apart from the central coordinatesof the respective pixels 341, or the information representing how muchfar the central coordinates of the respective sub-pixels 342 are locatedin average apart from the central coordinates of the pixels 341)representing the positions of the sub-pixels within the pixels in thedisplay portion 34 is stored in the memory portion 32 of the outputapparatus 12. That information, for example, is written to the memoryportion 32 at the time of the factory shipment or the like.

Then, when the control portion 31 of the output apparatus 12 receivesthe request for the information on the positions of the sub-pixels fromthe outside (such as the information processing apparatus 11), thecontrol portion 31 outputs the information on the positions of thesub-pixels stored in the memory portion 32 to the information processingapparatus 11 as the request source in response to the request concerned.

In this example, the information processing apparatus 11 acquires theinformation on the positions of the sub-pixels from the output apparatus12 in the stage, before the processing for producing the corrected imagedata is executed, at the timing at which the output apparatus 12 isconnected, or the like, and stores the information on the positions ofthe sub-pixels in the memory portion 22. The information processingapparatus 11 uses the information on the positions of the sub-pixels inthe processing for producing the corrected image data.

Incidentally, even when the information on the positions of thesub-pixels cannot be acquired even if the request is output to theoutput apparatus 12 in this example, the processing for producing thecorrected image data may be executed on the assumption that the centralcoordinates of all the sub-pixels are located at the centers of therespective pixels of the source image data. In this case, specifically,since the processing for image deformation which has been general in thepast can be used, a detailed description is omitted herein.

In addition, in the description until now, when the processing in whichthe assembly error of the displays or the lenses is taken intoconsideration is executed, the output apparatus 12 sends theinformation, on the assembly error of the lens 35, which had beenwritten at the time of the factory shipment to the informationprocessing apparatus 11 in response to the request. However, the presentembodiment is by no means limited thereto.

That is to say, in the present embodiment, the output apparatus 12 maybe provided with a device for measuring the position shift between theoptical axis of the lens 35 and the center of the corresponding displayportion 34. Specifically, an auto-collimator which is generally knownmay be used as this device. Incidentally, in the present embodiment, themeasurement of the assembly error may be one for measuring the positionshift with respect to an ideal optical axis, or may be one for measuringthe relative position shift between the optical axis of the lens and thecenter of the display portion 34.

In this example, instead of reading out the information on the assemblyerror stored in the memory portion 32 in response to the request for theinformation on the assembly error from the outside (such as theinformation processing apparatus 11), the control portion 31 of theoutput apparatus 12 measures the position shift between the optical axesof the lenses 35L and 35R, and the centers of the corresponding displayportions 34L and 34R, and outputs the information on the assembly errorthus measured to the information processing apparatus 11 or the like asthe request source.

REFERENCE SIGNS LIST

10 Information processing system, 11 Information processing apparatus,12 Output apparatus, 21, 31 Control portion, 22, 32 Memory portion, 23Interface portion, 33 I/O interface portion, 34 Display portion, 35Lens, 41 Source image receiving portion, 42 Corrected image producingportion, 43 Output portion, 341 Pixel, 342 Sub-pixel.

The invention claimed is:
 1. An information processing systemcomprising: an output apparatus provided with a display portionincluding a respective plurality of sub-pixels in each of a plurality ofpixels, the output apparatus serving to present an image displayed onthe display portion to a user through a lens; and an informationprocessing apparatus connected to the output apparatus serving toproduce an image to be displayed on the display portion, wherein theinformation processing apparatus includes a circuit operating to receivesource image data for the respective plurality of sub-pixels in each ofthe plurality of pixels of the display, a circuit operating to producecorrected image data obtained by correcting a distortion of the lensincluded in the output apparatus by carrying out a process on the sourceimage data for each of the plurality of pixels, including: (i) computinga central coordinate of a given one of the plurality of pixels as afunction of respective positions of each of the sub-pixels within thegiven one of the plurality of pixels of the source image data, (ii)obtaining a corresponding pixel of the source image data thatcorresponds to the central coordinate of the given one of the pluralityof pixels, through an arithmetic operation, and (iii) producing thecorrected image data by determining respective luminances of therespective plurality of sub-pixels of the given one of the plurality ofpixels based on one or more pixel values near the corresponding pixel ofthe source image, and a circuit operating to output the corrected imagedata produced to the output apparatus.
 2. An information processingapparatus provided with a display portion including a respectiveplurality of sub-pixels in each of a plurality of pixels, theinformation processing apparatus being connected to an output apparatuspresenting an image displayed on the display portion to a user through alens, and serving to produce an image to be displayed on the displayportion, the information processing apparatus comprising: a circuitoperating to receive source image data for the respective plurality ofsub-pixels in each of the plurality of pixels of the display; a circuitoperating to produce corrected image data obtained by correcting adistortion of the lens included in the output apparatus by carrying outa process on the source image data for each of the plurality of pixels,including: (i) computing a central coordinate of a given one of theplurality of pixels as a function of respective positions of each of thesub-pixels within the given one of the plurality of pixels of the sourceimage data, (ii) obtaining a corresponding pixel of the source imagedata that corresponds to the central coordinate of the given one of theplurality of pixels, through an arithmetic operation, and (iii)producing the corrected image data by determining respective luminancesof the respective plurality of sub-pixels of the given one of theplurality of pixels based on one or more pixel values near thecorresponding pixel of the source image; and a circuit operating tooutput the corrected image data produced to the output apparatus.
 3. Theinformation processing apparatus according to claim 2, wherein the pixelof the display portion included in the output apparatus includes aplurality of kinds of pixels in which positions of the sub-pixels aredifferent from one another, and the circuit operating to produceproduces the corrected image data with information representing anaverage position of positions for each sub-pixel of the plurality ofkinds of pixels as information on the positions of the sub-pixels.
 4. Anon-transitory, computer readable recording medium storing a computerprogram, which, when executed by an information processing apparatusprovided with a display portion including a respective plurality ofsub-pixels in each of a plurality of pixels, the information processingapparatus being connected to an output apparatus presenting an imagedisplayed on the display portion to a user through a lens, and servingto produce an image to be displayed on the display portion, causes theinformation processing apparatus to carry out actions, comprising:receiving source image data for the respective plurality of sub-pixelsin each of the plurality of pixels of the display; producing dataobtained by correcting a distortion of the lens included in the outputapparatus by carrying out a process on the source image data for each ofthe plurality of pixels, including: (i) computing a central coordinateof a given one of the plurality of pixels as a function of respectivepositions of each of the sub-pixels within the given one of theplurality of pixels of the source image data, (ii) obtaining acorresponding pixel of the source image data that corresponds to thecentral coordinate of the given one of the plurality of pixels, throughan arithmetic operation, and (iii) producing the corrected image data bydetermining respective luminances of the respective plurality ofsub-pixels of the given one of the plurality of pixels based on one ormore pixel values near the corresponding pixel of the source image; andoutputting the corrected image data produced to the output apparatus.